dhtmlxGrid. Step-by-step Guide. Big grid


by Ivan Petrenko

Introduction. What is this all about?

I've deleted a file recently. In a few moments I understood that it was a mistake (not the only one that day, to be honest). One would think: "What can be easier?" Just double-click the Recycle Bin and... I 've been waiting for 3 minutes, scrolling down, waiting again. My file was among thousands of others (right, I know it would be good to empty the Recycle Bin sometimes) and it took Windows (! - not even the web application, but Windows Explorer) minutes before I could scroll to the very bottom until I found my file. And then I thought: this can never happen with dhtmlxGrid!

Now I'm going to show you why.

Real Introduction.

Keeping  thousands of records in grid is a common requirement for most business applications. These records can be goods, names of employees, invoices, customers, some history records, etc. They can also be files for sure...  So, if you are developing a web application with a grid, you definitely need a grid control that is able to display...let's say... the whole list of you wife's purchases for the last two months, or the list of all possible reasons you can invent in order not to work on Sunday (which are going to be really long lists).

dhtmlxGrid is the very component you need!

This step-by-step tutorial will let you load 50,000 records into your grid and still have it working fast.

Firstly, let me outline a problem.

For example, you load a dataset with 10,000 records. Let's make some calculations.
It takes a second to load this dataset from the server to your browser (probably, it takes more or less time depending on your connection speed), half a second to convert it to grid internal format, and 0.01 second to draw each row in the grid (all these figures are approximate, but you get the idea, I hope).

The result of our calculation is 1+0.5+0.01*10000 = 101,5. It comes out more than a minute. So, you can go to the kitchen to make a cup of coffee... and even drink it if your computer is not fast enough. But why do you need to wait for 10,000 records to be drawn? We also think that you don't! You just need the first 100 rows to start working with as it is this number of rows that is visible in the grid frame. 

So we added some ingenious code
to the grid and called this new possibility "Smart Rendering", as our grid now needs to be smart enough to know which records to draw and which of them should be put off.

There are two variants of Smart Rendering in dhtmlxGrid:

Thus, for the first variant the format of XML remains the same as common grid, and it generally can be a static XML file:
<rows>
    <row id="xx">
        <cell>
        ...
        </cell>
    </row>
</rows>
For the variant with dynamic loading it gets two additional parameters and (as far as we should process some incoming arguments) needs to be created dynamically with some server side programming language:
<rows total_count="x" pos="y">
    <row id="xx">
        <cell>
        ...
        </cell>
    </row>
</rows>
Thus, this tutorial is not just about JavaScript, but also about server side a little. I'll show you server side code for PHP, ASP, JSP, ColdFusion. But let's do everything in its turn.
By the way, if some of the readers can send me necessary code for some other languages/technologies, I'll definitely put it here with big gratitude to the author.

Step 1. Include script and CSS files into the page

And again we start with including external JavaScript and CSS files into the page. In addition to the files we included in the previous chapter we'll use Smart Rendering extension to work with large amounts of data but still keep the grid fast. So additional file is ext/dhtmlxgrid_srnd.js:
<link rel="STYLESHEET" type="text/css" href="codebase/dhtmlxgrid.css">
<script src="codebase/dhtmlxcommon.js"></script>
<script src="codebase/dhtmlxgrid.js"></script>
<script src="codebase/dhtmlxgridcell.js"></script>
<script src="codebase/ext/dhtmlxgrid_srnd.js"></script>
<script>
    var gridQString = "";//we'll save here the last url with query string we used for loading grid (see step 5 for details)
    //we'll use this script block for functions
</script>

Step 2. Initialization of Grid with Smart Rendering

Depending on the data structure you'll need a grid with different columns set. I have 4 columns in the database table - unique ID (id), some name (nm), related alphanumeric code (code) and numeric value (num_val) (You can get the sql file here. It contains only 5,000 records to minimize its size).

This is an abstract sample, but we can think about those names as the names of some pharmaceuticals (it is not in reason to expect them to be something else), the codes will be their internal product codes, and numeric values will be the prices. So initialization code for such kind of a grid will be as follows:
   

<div id="products_grid" style="width:500px;height:200px;"></div>
<script>
    var mygrid = new dhtmlXGridObject('products_grid');

    mygrid.setImagePath("codebase/imgs/");

    mygrid.setHeader("Product Name,Internal Code,Price");

    mygrid.setInitWidths("*,150,150");

    mygrid.setColAlign("left,left,right");

    mygrid.setSkin("modern");

    mygrid.init();

    mygrid.enableSmartRendering(true);

</script>

As you probably remember from the previous chapter, I used body "onload" event to call grid initialization function. The code mentioned above is another case as it calls script methods on the page placing them after DIV container we want to place our grid into. The goal here is the same: to call the grid constructor after the DIV container was initialized.

So what is new in this script? A new line of code was added to enable Smart Rendering. As you see it is quite simple: just one command and you are ready to load thousands of records. Very simple indeed.

So here is what we have now:


Step 3. Loading Data. Server side support for Smart Rendering


As you already know (if you've read the Introduction) there are two variants of Smart Rendering in dhtmlxGrid. We'll concentrate on more complex one, which allows you working with much bigger datasets. I mean Smart Rendering with Dynamic Loading.

Below there are samples of server side code for creating an output XML based on incoming arguments and MySQL database for 4 most popular technologies. Load your variant of file with the following command (put it into script block right after mygrid.enableSmartRendering):
gridQString = "getGridRecords.php";//save query string to global variable (see step 5 for details)
mygrid.loadXML(gridQString );
Sending the request to the URL you specify inside loadXML method, grid adds two properties:


Thus GET request which comes to the server will look something like this: getGridRecords.php?posStart=199&count=100 .

Sample of server-side code. PHP.

<?php
    //set content type and xml tag
    header("Content-type:text/xml");
    print("<?xml version=\"1.0\"?>");
   
    //define variables from incoming values
    if(isset($_GET["posStart"]))
        $posStart = $_GET['posStart'];
    else
        $posStart = 0;
    if(isset($_GET["count"]))
        $count = $_GET['count'];
    else
        $count = 100;
   
    //connect to database
    $link = mysql_pconnect("localhost", "user", "pwd");
    $db = mysql_select_db ("sampleDB");
   
    //create query to products table
    $sql = "SELECT  * FROM products";
   
    //if this is the first query - get total number of records in the query result
    if($posStart==0){
        $sqlCount = "Select count(*) as cnt from ($sql) as tbl";
        $resCount = mysql_query ($sqlCount);
        $rowCount=mysql_fetch_array($resCount);
        $totalCount = $rowCount["cnt"];
    }
   
    //add limits to query to get only rows necessary for the output
    $sql.= " LIMIT ".$posStart.",".$count;
   
    //query database to retrieve necessary block of data
    $res = mysql_query ($sql);

    //output data in XML format   
    print("<rows total_count='".$totalCount."' pos='".$posStart."'>");   
    while($row=mysql_fetch_array($res)){
        print("<row id='".$row['id']."'>");
            print("<cell>");
                print($row['nm']);  //value for product name
            print("</cell>");
            print("<cell>");
                print($row['code']);  //value for internal code
            print("</cell>");
            print("<cell>");
                print($row['num_val']);    //value for price
            print("</cell>");
         print("</row>");
    }
    print("</rows>");
?>

* - sample code was simplified to concentrate you on the main commands. Some necessary error handlers etc. were omitted.

Sample of server side code. JSP.

<%@ page import = "java.sql.*" %>
<%
    String db_ipp_addr = "localhost";
    String db_username = "root";
    String db_password = "1";
    String db_name = "sampleDB";
   
    // set content type and xml tag
    response.setContentType("text/xml");
    out.println("<?xml version=\"1.0\" encoding=\"UTF-8\"?>");

    // define variables from incoming values
    String posStart = "";
    if (request.getParameter("posStart") != null){
        posStart = request.getParameter("posStart");
    }else{
        posStart = "0";
    }   
   
    String count = "";
    if (request.getParameter("count") != null){
        count = request.getParameter("count");
    }else{
        count = "100";   
    }   
   
    // connect to database
    Connection connection = null;
    Statement statement = null;
    ResultSet rs = null;
    String connectionURL = "jdbc:mysql://" + db_ipp_addr + ":3306/" + db_name;
   
    Class.forName("com.mysql.jdbc.Driver").newInstance();
    connection = DriverManager.getConnection(connectionURL, db_username, db_password);

    // query to products table
    String sql = "SELECT  * FROM products";

    // if this is the first query - get total number of records in the query result
    String totalCount = "";
    if (posStart.equals("0")){
        String sqlCount = "Select count(*) as cnt from (" + sql + ") as tbl";
        statement = connection.createStatement();
        rs = statement.executeQuery(sqlCount);
        rs.next();
        totalCount = rs.getString("cnt");
        rs.close();
    } else {
        totalCount = "";
    }

    // add limits to query to get only rows necessary for output
    sql += " LIMIT " + posStart + "," + count;

    // Execute the query
    statement = connection.createStatement();
    rs = statement.executeQuery(sql);
   
    // output data in XML format  
    out.println("<rows total_count='" + totalCount + "' pos='" + posStart + "'>");
    while (rs.next()) {
        out.println("<row id='" + rs.getString("id") + "'>");
            out.println("<cell>");
                out.println(rs.getString("nm"));  // value for product name
            out.println("</cell>");
            out.println("<cell>");
                out.println(rs.getString("code")); // value for internal code
            out.println("</cell>");
            out.println("<cell>");
                out.println(rs.getString("num_val"));   // value for price
            out.println("</cell>");
        out.println("</row>");
    }
    out.write("</rows>");
    rs.close();
%>

Sample of server side code. ASP

<%@ LANGUAGE = VBScript %>
<% option explicit %>

<%
    Dim db_ipp_addr, db_username, db_password, db_name
    db_ipp_addr = "localhost"
    db_username = "root"
    db_password = "1"
    db_name = "sampleDB"
   
    ' set content type and xml tag
    Response.ContentType = "text/xml"
    Response.write("<?xml version=""1.0"" encoding=""UTF-8""?>")

    ' define variables from incoming values
    Dim posStart, count
    If not isEmpty(Request.QueryString("posStart")) Then
        posStart = Request.QueryString("posStart")
    Else
        posStart = 0
    End If   

    If not isEmpty(Request.QueryString("count")) Then
        count = Request.QueryString("count")
    Else
        count = 100
    End If   
           
    ' connect to database
    Dim objConnection, rs, connString, sql
    Set objConnection = Server.CreateObject("ADODB.Connection")
    Set rs = Server.CreateObject("ADODB.Recordset")
    connString = "DRIVER={MySQL ODBC 3.51 Driver}; SERVER=" & db_ipp_addr & "; DATABASE=" & db_name & "; UID=" & db_username & "; PWD=" & db_password
    objConnection.Open connString
   
   
    ' query to products table
    sql = "SELECT  * FROM products"

    ' if this is the first query - get total number of records in the query result
    Dim sqlCount, totalCount
    If posStart = 0 Then
        sqlCount = "Select count(*) as cnt from (" & sql & ") as tbl"
        rs.Open sqlCount, objConnection
        totalCount = rs("cnt")
        rs.Close
    Else
        totalCount = ""   
    End If

    ' add limits to query to get only rows necessary for output
    sql = sql & " LIMIT " & posStart & "," & count

    ' Execute the query
    rs.Open sql, objConnection
   
    ' output data in XML format  
    Response.write("<rows total_count='" & totalCount & "' pos='" & posStart & "'>")
    Do while not rs.EOF
        Response.write("<row id='" & rs("id") & "'>")
            Response.write("<cell>")
                Response.write(rs("nm"))  ' value for product name
            Response.write("</cell>")
            Response.write("<cell>")
                Response.write(rs("code"))  ' value for internal code
            Response.write("</cell>")
            Response.write("<cell>")
                Response.write(rs("num_val"))    ' value for price
            Response.write("</cell>")
        Response.write("</row>")
        rs.MoveNext
    Loop
    Response.write("</rows>")
    rs.Close
    Set rs = Nothing
    objConnection.Close
    Set objConnection = Nothing
%>

Sample of server side code. Cold Fusion.

<cfset dsn = "sampleDB">
<cfsetting  enablecfoutputonly="yes">

<!--- set content type and xml tag --->
<cfcontent reset="yes" type="text/xml; charset=UTF-8"><cfoutput><?xml version="1.0"?></cfoutput>

    <!---    define variables from incoming values --->
    <cfif isDefined("url.posStart")>
        <cfset posStart = url.posStart>
    <cfelse>
        <cfset posStart = 0>
    </cfif>   

    <cfif isDefined("url.count")>
        <cfset count = url.count>
    <cfelse>
        <cfset count = 100>
    </cfif>   

    <!--- if this is the first query - get total number of records in the query result --->
    <cfif posStart eq 0>
        <cfquery datasource="#dsn#" name="getCount">
            Select count(*) as cnt
            FROM products
        </cfquery>
        <cfset totalCount = getCount.cnt>
    <cfelse>
        <Cfset totalCount = "">   
    </cfif>

    <!--- query to products table --->
    <cfquery datasource="#dsn#" name="getRecords">
        SELECT  *
        FROM products
        <!--- add limits to query to get only rows necessary for output --->
        LIMIT #posStart#, #count#
    </cfquery>
   
    <!--- output data in XML format   --->
    <cfoutput><rows total_count="#totalCount#" pos="#posStart#"></cfoutput>  
    <cfloop query="getRecords">
        <cfoutput><row id="#getRecords.id#"></cfoutput>
            <!--- value for product name --->
            <cfoutput><cell>#getRecords.nm#</cell></cfoutput>
            <!--- value for internal code --->
            <cfoutput><cell>#getRecords.code#</cell></cfoutput>
            <!--- value for price --->
            <cfoutput><cell>#getRecords.num_val#</cell></cfoutput>
        <cfoutput></row></cfoutput>
    </cfloop>
    <cfoutput></rows></cfoutput>


After loading the file the grid will look something like in the picture below. When you stop scrolling the grid will load and draw a new portion of records.


Step 4. Filtering. Passing additional parameters to server side

Why do you think you need additional parameters and why am I talking about them together with Smart Rendering? There are two situations which come to my mind right away:


Let's add the possibility to filter grid data by product name mask. Put the following code right before the DIV container we used for grid initialization:

<input type="Text" id="nm_mask">
<input type="Button" value="Filter" onclick="applyFilter()">
As you see, clicking the "Filter" button we call applyFilter function, which doesn't exist as we haven't created it yet. Let's do it now. It will contain the magic of getting the content for grid based on additional parameter :)
function applyFilter(){
    mygrid.clearAll(); //remove all data
    gridQString = "getGridRecords.php?name_mask="+document.getElementById("nm_mask").value;//save query string in global variable (see step 5 for details)
    mygrid.loadXML(gridQString); // load new dataset from sever with additional parameter passed
}
Put it into the script block we left for functions (or any other - it doesn't matter).
Now getGridRecords file gets additional parameter named name_mask, which you can add to the query to filter the results by name. For example (PHP):

    //query to products table
    $sql = "SELECT  * FROM products";
    if(isset($_GET["name_mask"]))
        $sql.=" Where nm like '".$_GET["name_mask"]."%'";

Step 5. Server Side Sorting.

When we have 50,000 records in grid or just going to have but do not really have (as we work with Smart Rendering, with Dynamic Loading) client side sorting will not help a lot. Our grid just doesn't know all the values yet. So we need to move sorting to server side. How? Just relax. It is really simple.

There will be 3 stages of the plan:
  1. Cancel client side sorting;
  2. Clear grid and load the data sorted on server side;
  3. Set position and direction of a marker in the grid header to show sorting direction.

To cancel client side sorting we need to enable sorting for the columns first. Like it was described in the previous tutorial, you can do this with setColSorting method, but the sorting type for all three columns will be "server":
mygrid.setColSorting("server,server,server");
The "server" sorting type means nothing for sorting routine of the grid, so it'll just ignore it. This information is mostly for myself as column sorting was moved to server side. Put this command somewhere before mygrid.init() .
Now we are ready to execute stages 1 and 2 of the plan. We'll do this with onBeforeSorting event handler. First let's define the handler function. According to events documentation it gets 3 incoming arguments:

And this function will work IN PROFESSIONAL EDITION of the grid. Only. Sorry, from now on only those who "got the tickets" go further...
 
OK, if you are still reading, here is the complete code for the event handler function. Put it into the script block we've left for functions (or in any other one):

    function sortGridOnServer(ind,gridObj,direct){
            mygrid.clearAll();
            mygrid.loadXML(gridQString+(gridQString.indexOf("?")>=0?"&":"?")+"orderby="+ind+"&direct="+direct);
            mygrid.setSortImgState(true,ind,direct);
            return false;
        }
The above mentioned code does the following:

  1. Removes all rows from grid;
  2. Loads new content passing column index as orderby parameter and order direction (asc for ASC or des for DESC) as direct parameter. Here it becomes clear why we needed to save gridQString in the previous steps - because we need to sort exactly the same content which we got last time we loaded grid.
  3. Sets Sort Image visible for the column we sort and with the direction we need.
  4. Cancels client side sorting by returning "false" from event handler function for onBeforeSorting event. This is what we meant in point 1 of the plan.

Now when we have sortGridOnServer function, we can add the following command to set the event handler to grid initialization script:
mygrid.attachEvent("onBeforeSorting",sortGridOnServer);
Server side changes are simple, as we just add Order by statement for the field in table that corresponds to a column in grid, and set the direction: ASC if asc, DESC if des.
Here is a code sample for PHP:
    //order by
    $columns = array("nm","code","num_val");
    if(isset($_GET["orderby"])){
        if($_GET["direct"]=='des')
            $direct = "DESC";
        else   
            $direct = "ASC";
        $sql.=" Order by ".$columns[$_GET["orderby"]]." ".$direct;
    }

Download related files